1. Field of the Invention
Embodiments of the invention generally relate to consumer electronics devices. More specifically, embodiments of the invention relate to techniques for performing dynamic randomized controlled testing for design choices in consumer electronics devices.
2. Description of the Related Art
Randomized controlled testing (RCT) refers to an experimental methodology frequently used to test the efficacy or effectiveness of something, e.g., a pharmaceutical compound. In a randomized controlled trial, one or more test groups of users are compared to one or more control groups of users. Typically, qualified participants are randomly allocated into groups, where participants in some groups receive the item or items of interest while participants in other groups do not. Importantly, the participants are generally unaware of which group they are assigned to. So long as the number of participants is sufficient, RCT provides an effective method for balancing out random or noise factors between groups, allowing the aspect of the item of interest to be evaluated quantitatively against the control group.
Design choices for consumer electronics (CE) devices, however, have generally not been the subject of randomized controlled testing as CE devices are generally manufactured and distributed using a single, fixed form. This has occurred as building and distributing multiple versions of a given CE device is hard to manage. For example, distributing multiple versions randomly (i.e. without a preponderance of one version in a particular channel) has proven very difficult. Further, a manufacturer or distributor of a particular CE device may be reluctant to distribute different versions of a single product, particularly where versions of an untested one may in practice prove to be highly disfavored by consumers.
Additionally, even were a manufacturer to distribute multiple CE devices, collecting feedback on usage and results is extremely limited, since there is not a reliable means for consumers' behavior to be observed, monitored, or tested. While a survey may result in some consumer feedback, this approach has both response and bias issues. Further still, a manufacturer may be unable to discern which version of a given CE device a consumer is responding about.